Period-two cycles in a feed-forward layered neural network model with symmetric 

sequence processing 
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The effects of dominant sequentiai interactions are investigated in an exactly solvable feed-forward 
layered neural network model of binary units and patterns near saturation in which the interaction 
consists of a Hebbian part and a symmetric sequential term. Phase diagrams of stationary states 
are obtained and a new phase of cyclic correlated states of period two is found for a weak Hebbian 
term, independently of the number of condensed patterns c. 
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I. INTRODUCTION 

The dynamics and the stationary states of recurrent 
attractor neural networks that process sequences of pat- 
terns have been studied over some time and there has 
been a recent revival of interest near the storage sat- 
uration limit in large networks [1-8]. Either a process 
involving a sequence of patterns [3-7] (referred to in this 
paper as asymmetric sequence processing), leading to a 
stationary limit cycle, or a process involving a pair of 
sequences, one with patterns in increasing order and the 
other one with patterns in decreasing order (referred to 
as symmetric sequence processing in what follows), were 
considered in those works. 

Symmetric sequence processing competing with pat- 
tern reconstruction favored by a Hebbian term of fixed 
strength has also been considered [Ei]. The ratio J H /Js 
between the strengths Jh and Js of the Hebbian term 
and of the pair of sequences, respectively, has been re- 
stricted to a mostly dominant Hebbian strength, that is 
to 1 < Jh/Js < oo, leading to phase diagrams which 
only exhibit fixed-point solutions including non-trivial 
correlated attractors [l], Q. These are states that in- 
dicate a selectivity in response to a set of previously 
learnt patterns in which the correlation coefficients for 
the attractors with increasingly distant patterns from a 
stimulus are decreasing functions which eventually be- 
come vanishingly small. The sequential part of the in- 
teraction induces transitions between patterns, whereas 
a sufficiently strong Hebbian term locks the transitions 
favoring single-pattern recognition. The case of network 
models with a weak Hebbian interaction competing with 
a dominating symmetric sequential processing in which 
Jh/Js < 1 has, apparently, not been studied before and 
it is important to find out what kind of solutions appear 
in that case and their possible biological implications. 

Indeed, the connection between the information input 
in a network of contiguous stimuli in a training sequence 
of uncorrelated patterns and correlated delay activity as 
an output has been of great interest to explain the results 
of experimental recordings of a visual-memory task in the 
infcrotcmporal (IT) cortex of monkeys [8-13], in which 



correlated states play an essential role. The results can 
be interpreted as a connection between persistent cortex 
activity and long-term associative memory described by 
a fixed-point attractor dynamics, and the early model 
calculations that have been done are based on the com- 
petition between pattern reconstruction and symmetric 
sequence processing 0, [ll| . 

There could be other attractors that may predict fur- 
ther behavior on visual-memory in the primate IT cor- 
tex viewed as a dynamical system, generating either limit 
cycles, chaotic or other kinds of behavior. Earlier model 
calculations on the competition between pattern recon- 
struction and asymmetric sequence processing exhibit 
periodic and stationar y f ixed-point attractors, besides 
quasi-stationary states \l4 . [H| . Motivated by the results 
that appeared in the type of experiments on the IT cortex 
in monkeys and their interpretation [1,8-13], it would be 
interesting to see if there are periodic attractors that ap- 
pear as correlated states in a neural network model with 
the kind of competition between pattern reconstruction 
and symmetric sequence processin g th at has been stud- 
ied before in a recurrent network (Hlll| , now with a weak 
Hebbian term. The presence of periodic attractors would 
suggest a new kind of persistent oscillating activity in the 
IT cortex. The main purpose of this work is to explore 
this issue and with that in mind one has to resort to 
a dynamical procedure even to detect stationary cyclic 
behavior and to determine the emergent properties. 

Since the dynamics of fully recurrent neural networks 
with binary units is already fairly complicated, and even 
more so with graded-response or other more realistic 
units, we make a drastic simplification to start with in 
order to get to the essentials of our issue. We are mainly 
interested in finding out if there are cyclic attractors and 
in characterizing their properties for a sizeable critical 
storage of patterns. Our interest is in the role of com- 
peting interactions that favor either transitions between 
patterns or the recognition of specific patterns. To that 
end we focus on a simple attractor feed-forward layered 
network model of binary units and patterns with a par- 
allel dynamics and without lateral connections (l6j . The 
model is exactly solvable and has been extensively used 
in the past as a model for associative memory that has 
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all the stationary features of a recurrent network and it is 
particularly suited to detect stationary non-equilibrium 
states. We show that, despite sequential learning, the 
procedure involves in practice a finite number of recur- 
sion relations even for a macroscopically large system. 
The effect of the lateral connections is to change the 
quantitative results of the model. 

To make that point clear we construct the phase di- 
agrams that describe the network behavior for an arbi- 
trary strength of the Hcbbian term in order to check first 
that we get the same qualitative behavior as that already 
found for a recurrent network in the case of balanced in- 
teractions or for a dominant Hebbian term. Having stud- 
ied that part, one can be reasonably confident that the 
behavior of the layered network for dominant sequential 
interaction should also describe the correct qualitative 
behavior of a recurrent network. The fully quantitative 
behavior of that network is a relevant issue which is be- 
yond the scope of the present paper. 

It will be shown that a phase of cyclic stationary so- 
lutions of period two is obtained, independently of the 
number of condensed patterns c with macroscopic overlap 
with the states of the network, and that this is a phase of 
non-trivial correlated states. These are states that could 
reflect a new kind of persistent activity in visual-memory 
task experiments. On the other hand, the stability of the 
cyclic phase is strongly dependent on c. These features 
distinguish the properties of the present model from those 
for asymmetric sequence processing competing with pat- 
tern reconstruction where cycles of period c appear, for 
arbitrary c [l5j . 

The purpose of studying a simplified model which has 
the qualitative features of a recurrent network with more 
realistic interactions is that it tells what to look for and 
what could be relevant in a more biological minded net- 
work. The paper is organized as follows. In Sec. 2 we 
present the model and outline the derivation of the dy- 
namics with the aid of an appendix. We present our main 
results in Sec. 3 and conclude with a further discussion 
in Sec. 4. 



II. THE MODEL AND THE DYNAMICS 

The network model consists of L layers, each contain- 
ing N binary units (neurons) in states Si(l) = ±1, where 
i denotes the unit and I the layer. The state +1 repre- 
sents a firing unit and the state —la unit at rest. The 
state of each unit on a given layer is determined in paral- 
lel by the state of all units on the previous layer, the layer 
label acting as a time step, according to the stochastic 
rule with probability [l6| 

m(i+i)\s(i)) = cxp J^lty: ( !,t 1)] , (i) 



2cosh[/?/i i (Z + 1))] 



N 



hi(i + i) = ^Jij^SjU) , 



(2) 



where hi(l + l) is the local field at unit i on layer l + l due 
to the set of states S(l) of all units on layer I and Jij(l) is 
the synaptic coupling between unit j on layer I and unit i 
on layer l + l. There is no feedback in the updating of the 
units and the first layer has to be set externally in a given 
state. The parameter (3 = T _1 controls the synaptic 
noise such that the dynamics is fully deterministic when 
T — > and fully random when T — > oo. 

A macroscopic set of p = aN statistically independent 
and identically distributed random patterns {^(Oli f 1 ~ 
1, .. . ,p, with components (I) = ±1 and probability \ 
for either value, are stored on every layer independently 
of other layers, according to the learning rule 



Mi) 



i 

N 



(3) 



Thus, there are only interactions between pairs of units 
on consecutive layers. Connections between units on 
more distant layers, as well as lateral connections be- 
tween units on the same layer are excluded. Here, X^ p 
are the elements of the matrix 



X = 



A 
B 



and 



\ P = vb^ p + (1 - v) {S^ p+1 +<^ i/} _i) , (4) 
B w = bS^ p + (1 - b) (S^p+i + f^.p-i) , 

are the elements of the c x c and (p — c) x (p — c) blocks 
A and B responsible for the signal and for the noise 
in the local field, respectively, and c is the number of 
condensed patterns that yield macroscopic overlaps de- 
fined below. The diagonal two-block interaction ma- 
trix reflects the fact that the patterns are associated in 
two independent cycles, one for the condensed patterns 
(i c+1 {l) = £ l U)) and the other one for the non-condensed 



patterns (£ p (I) = £ (/)). This guarantees the appli- 
cability of the procedure. The first parts of A and B 
contribute to a Hcbbian interaction Jh and their second 
parts contribute to the symmetric sequential interaction 
Js- The training of the network model may be thought 
to proceed in two stages assuming the patterns are num- 
bered in a given order. In one stage the set of patterns is 
presented to the network in random order, every pattern 
being presented the same number of times. This builds 
up the Hebbian part of the learning rule, whereas the 
sequential part of the rule takes place as follows in an- 
other stage. The patterns are ordered in two sequences, 
one sequence in increasing order in which each pattern 
is presented with the following pattern in the sequence, 
and the other sequence in decreasing order where every 
pattern is presented with the previous pattern. 

The crucial parameter is v determining the ratio 
Jh/Js of the Hebbian to sequential interaction in the 
signal term of the local field. On the other hand, differ- 
ent sequential noise levels given by b should yield quali- 
tatively similar results as found in previous works [J, [l5| . 
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This does not mean that 6 is an irrelevant parameter and 
we consider this point below. When 6=1 there is a 
purely Hebbian noise and for any other b there is a Hcb- 
bian plus sequential noise. In distinction to other works, 
the choice (0 < u, b < 1) enables us to explore the full 
range of parameters v and b. 

The macroscopic overlap components m^{l) of 0(1), 
between the configuration S(l) and the condensed pat- 
terns, are given by the large- N limit of 

1 N 

"*ft(0 = jyf£tf(W(f)) ; m = i,...,c, (5) 

i=l 

where (...) denotes a thermal average with Eq. (fTJ), 
whereas the overlaps with the remaining (p — c) non- 
condensed patterns are M^(l) = 0(1/VN). 

Following the standard procedure for the layered net- 
work, one may write the local field as a sum of a signal 
and a noise term tOi(l) due to the condensed and the non- 
condensed patterns, respectively [l^ . The noise follows 
a Gaussian distribution with mean zero and a variance 
A 2 (Z) given by the large-N limit of 

A^(0= E (W), (6) 

fj.=c+l 

where the overbar denotes the average over all the pat- 
terns, in which 

Q%{1) = bM*(l) + (1 - b)[M£-\l) + M» + \l)] . (7) 

The non-condensed overlaps that appear here depend on 
all patterns as functions of the full local field. Averages 
over the non-condensed patterns can then be performed 
by integration and we obtain first the recursion relations 
for the macroscopic overlaps m(l) = (mi(l), . . . , m c (l)) 

m(Z + l) = (£ J Dz t&nh{/3[£.Am(l)+A(l)z]})£ , (8) 

where Dz = e~ z l 2 dz /\/2tt and (••■}£ denotes an ex- 
plicit average over the condensed patterns which does 
not depend on the specific realization of the patterns. 

To obtain a dynamic equation for A 2 (Z) we need re- 
cursion relations not only for the average squared non- 
condensed overlaps, (M^(l) 2 ) and (M^ 1 (I) 2 ) , which 
can be derived in the usual way [161 ]. but also 
for the correlation of two consecutive overlaps and 

two next-to-consecutive overlaps, (M^(/)Af^ ±1 (/)) and 

(M^ _1 (/)M^ +1 (Z)), respectively. These generate, in 
turn, correlations of overlaps between more distant pat- 
terns, which requires to keep track of a general form [l5j 

c 2 n {i) = E (W)QT®) > ( 9 ) 

in which n = 0, . . . ,p — c — 1 and C p _ c (Z) = A(Z). The 
p — c recursion relations for the C n 's can be obtained in 



a systematic way, as outlined in the appendix, leading to 
a variance of the noise which depends on the spin-glass 
order parameter q(l) = (S(l)) 2 , 

q{l) = { J Dz tanh 2 {/3[£.Am(Z) + A{l)z)})^ . (10) 

In the case where 6=1 (Hebbian noise), the variance 
follows the simple form A 2 (Z + 1) = a + K 2 (l)A 2 (l), 
where K{1) = [3{1 — q{l))- In the case of the absence 
of stochastic noise (a = 0), the equations coincide with 
those for a recurrent network with a parallel dynamics. 
The main difference between the layered and the recur- 
rent network is the absence of recurrent connections in 
the former. The recurrent connections tend to amplify 
the effects produced by the stochastic noise and the fact 
that the equations are the same when a = is simply 
due to the absence of noise to amplify in the recurrent 
network in that case. In the absence of stochastic noise 
the equations are also the same for the recurrent network 
with either symmetric or asymmetric extreme dilution 

a. 

Thus, the network dynamics is described by the recur- 
rence relations for the vector overlap ((5|) and for all the 
C„'s that go into the variance of the noise. Although 
the equations form an infinite set in the p — > oo limit, 
the number of significant C n 's is finite making the model 
solvable in practice. The transients and the dynamic evo- 
lution of the network can be studied in full detail but here 
we restrict ourselves to the stationary states. 



III. RESULTS 

Consider first the (T, v) phase diagram of stationary 
states for a = and various numbers of condensed 
patterns c, shown in Fig. 1(a). The Hopficld ansatz 

m A«(l) = <V-i' f° r f 1 = 1; ■ • ■ i c j i s used as initial condi- 
tion. The states corresponding to fixed-point solutions 
and the phase boundaries for v > 0.5 are precisely the 
same as those found for a recurrent network [TJ, since 
the equations for the order parameters are exactly the 
same in the layered and in the recurrent network when 
a = 0. The Hopfield-like states (H) have one large con- 
densed overlap component and the others are either small 
or zero. The phase of symmetric-like states (S) has equal 
or nearly equal overlap components, at high or low T, 
respectively, ending at a paramagnetic phase (P), with 
m = and q = 0. There is also a phase of non-trivially 
correlated states (D) in which the correlation coefficients 
for the overlaps with increasingly distant patterns from 
a stimulus become gradually smaller, as will be seen be- 
low. The phase boundaries for v > 0.5 arc practically 
independent of c. 

The novelty in the phase diagrams are stationary cyclic 
solutions (C) of period two, for any c, in the region 
< < 0.5 in which m(l + 2) = m(l). These are 
the only stable states below the phase boundaries for the 
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FIG. 1: Phase diagrams (a) for a — and (b) T = with a 
purely Hebbian noise (6 = 1). The dotted and full lines in- 
dicate, respectively, continuous and discontinuous transitions 
and the phases are described in the text. Some points on the 
phase boundary for c = 22 are indicated with crosses. 



presence of cyclic states. The main difference with asym- 
metric sequence processing studied in an earlier work [l5j 
is the presence in that case of cycles of period c, as well as 
quasi-periodic states. Fig. 1 now shows that the size of 
the cyclic phase decreases or increases with an increase 
of c, if c is even or odd, respectively, and there are no 
cyclic solutions for c < 7 in the latter case. These prop- 
erties have been checked by a linear stability analysis of 
the S phase for a = in extension of earlier work [lj| . 
Clearly, the periodic solutions are fairly robust to synap- 
tic noise T. In relation to a recent work (jj, we also 
studied our model with a finite number of independent 
pairs of sequences for a = and low v and found only 
cyclic solutions of period two, for any number of stored 
sequence pairs and for any c in each sequence. 

The effects of stochastic noise due to a macroscopic 
number of patterns p = aN are shown in Fig. 1(b) for 
T = 0, b = 1 (Hebbian noise) and various c. For the non- 
condensed overlaps we choose the initial C^ (l) = a, for 



all n, and for the condensed overlaps we take again a Hop- 
field initial condition. As usual in the layered network, 
there is now a spin-glass phase with q ^ (labeled SG). 
For v = 1 we recover the critical storage ratio a c ~ 0.269 
for the Hebbian layered network model [l6| . Also here we 
find the same kind of stationary states as in Fig. 1 (a) for 
v > 0.5 and stationary cyclic states of period two in the 
region of low with the absence of the latter for c < 7 in 
the case of odd c. Again, the boundaries between phases 
of fixed-point states are fairly independent of c, while 
the cyclic phase boundaries have a similar dependence 
on c as in the (T, v) phase diagram. We also found that 
the cyclic phase boundaries almost do not vary beyond 
c = 11 and c = 22, for odd and even c, respectively. They 
are very close and they should be independent of c in the 
large c limit. 




0.2 0.4 0.6 0.8 



b 

FIG. 2: Critical storage ratio as a function of 6 for T = and 
c — 13 in the Hopfield-like phase (y = 1 and v = 0.9) and in 
the cyclic phase (y = and v = 0.01). 

Although Fig. 1 gives a fair account of the phase dia- 
gram for Hebbian noise (b = 1), one may ask which is the 
effect of other noise parameters (b ^ 1) and in Fig. 2 we 
show the critical values a c for the existence of two typical 
states in each of the phases H and C as functions of b, 
for c = 13 and T = 0. In the case of fixed-point states, 
that is within the phase H (and also for the D phase, not 
shown in the figure) , a c increases with increasing v for a 
given b whereas in the case of cyclic states a c decreases, 
as one would expect. Similar results are obtained for 
c = 12. There is a maximum a c for an optimal b ~ 0.748 
and the kind of noise becomes more relevant for a dom- 
inant sequential part (small v) in the signal of the local 
field. 

We consider next the solutions for the stationary over- 
laps that describe the long-time behavior of the network, 
for a weak Hebbian term. The overlaps for states in the 
phase diagrams bifurcate from a fixed-point behavior in 
the S phase to stable stationary limit cycles on the con- 
tinuous (discontinuous) transition from the S to the C 
phase, for even (odd) c, respectively. In order to de- 
scribe one of our main results, that is the nature of the 
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cyclic behavior we illustrate this for the overlaps in Fig. 
3 for c = 13 when a = for a typical low synaptic noise 
of T = 0.3. The bifurcation diagram contains the sta- 
tionary values of the first seven overlaps as functions of 
v, with 171^(1) = (5^1 as an initial condition. Each over- 
lap assumes a larger and a smaller value in the cyclic 
phase marked with the same symbols (for clarity, only 
the larger one is labeled), with a decreasing oscillation 
amplitude as we move away from the stimulated pattern. 
For quite higher noise levels, say T = 1.25 and appropri- 
ate values of v (see below), all the pairs of overlap compo- 
nents keep oscillating between the same upper and lower 
values. Thus, for such high levels of T, the cyclic phase 
is already a phase of a pair of symmetric states, one with 
all equal larger condensed overlaps and the other with all 
equal smaller overlaps. In addition, the overlap compo- 
nents have the symmetry m^ l+n (t) = m^_„(t), where fi 
is the stimulated pattern and n = 1, . . . , (c — l)/2. 
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and a distance d = \f/, — v\ away, defined here as 
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FIG. 3: Overlaps for the stationary cycles of period two, dis- 
cussed in the text, that bifurcate from the symmetric phase 
for c = 13, a = and T = 0.3. 



In the case of even c, there is first a continuous transi- 
tion to a pair of overlaps with decreasing v such that all 
solutions keep oscillating between an upper and a lower 
value. There is a further, discontinuous transition, for 
lower values of is, to distinct pairs of overlaps that is 
similar to the discontinuous transition for the case of 
odd c, with the same symmetry between overlaps now 
for n = 1, . . . , (c — 2)/2. Thus, the behavior of the over- 
lap components has a rich structure which depends on 
whether c is even or odd. 

In order to demonstrate that the cyclic states may be 
non-trivial correlated attractors for low T and v, we con- 
sider next the correlation coefficients between the attrac- 
tors corresponding to any two condensed patterns, £ M 



C d 



)«<)-«)) 



(tanh(/3^)tanh(/3/i ,y )) < 
(tanh 2 (/3/^))t 



(11) 



where (a?) is the attractor corresponding to the initially 
stimulated pattern p and (erf) is its mean value over the 
network which is zero for the unbiased patterns we are 
using here. 

We show in Fig. 4 the dependence of the correlation co- 
efficients for the states corresponding to the larger over- 
lap component of each pair of states with the distance 
d to increasingly distant condensed patterns in the se- 
quence from a given stimulated one, for c — 13 and a = 0, 
that is, in the absence of stochastic noise where the re- 
sults are the same for the layered and the recurrent net- 
work. We do this, as indicated, for (T, v) = (0.3, 0.01) 
and also for (1.25,0.001) both within the phase of cyclic 
states. In the first case, where each overlap component 
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FIG. 4: Correlation coefficients between attractors as a func- 
tion of the distance d from a reference pattern, defined in the 
text, for c = 13 and a = in the phases C (two upper curves) 
and D (lower curve). The lines are a guide to the eye. 

assumes distinct larger and smaller values, there are also 
correlation coefficients for the smaller values, not shown 
in the figure, which turn out to decrease less rapidly than 
the coefficients for the larger values. The reason for this 
is that there is a weaker distinction between the smaller 
components than among the larger ones. For the higher 
T = 1.25 the correlation coefficients are already indepen- 
dent of d due to the fact that all the pairs of overlap 
components are oscillating between the same upper and 
lower values. 

In distinction to the fixed-point correlated states in 
phase D, shown for our model by the lower set of results 
in the figure for T = 0.03 and v = 0.625, the correla- 
tion coefficients for the cyclic states do not decrease to 
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zero and. instead, exhibit a behavior typical of quasi- 
symmetric states for low T. 



IV. DISCUSSION 

To summarize our results, we obtained a closed-form 
attractor dynamics for a feed-forward layered network 
model of binary units and patterns in terms of a finite 
number of macroscopic order parameters for the com- 
petition between pattern reconstruction and symmetric 
sequence processing. The dynamics is a parallel one, in 
which all units in each layer are updated simultaneously, 
and the work presented here is restricted to the station- 
ary states of the dynamics which exhibits either fixed- 
point or cyclic behavior of period two, depending on the 
relative strength of the interactions. Either kind of be- 
havior is an emergent property of the network which is 
an outcome of the dynamics. 

Full phase diagrams of stationary network behavior 
were obtained, either for a finite loading of patterns or in 
the saturation limit for the storage of a macroscopic num- 
ber of patterns. In the case of a balanced or dominant 
Hcbbian term, that is for v > 0.5, we obtained qualita- 
tively the same phase diagrams as those found in work 
by previous authors for a recurrent attractor network ex- 
hibiting Hopficld-like states, correlated states and sym- 
metric ordered states [l[ . This suggests that the layered- 
network dynamics discussed here may also serve to make 
qualitative predictions about a recurrent network in the 
case of a weak Hebbian term. Of course, to get the right 
quantitative behavior expected for a recurrent network in 
the storage saturation limit, and for detailed comparison 
with experiments, one has to start by including lateral 
connections between units in a layer but this involves a 
more complicated dynamics which is beyond the scope 
of this paper. For v below 0.5 we have first a regime 
of symmetric fixed-point behavior and for smaller val- 
ues of v we find the cyclic attractors discussed in this 
work. These are states that have decreasing correlation 
coefficients with increasingly distant attractors from an 
initially stimulated pattern for a sizeable range of synap- 
tic noise. 

The stationary overlaps between the states of the net- 
work and the condensed patterns describe the long-time 
behavior of the system. As we saw, the overlaps for states 
within the cyclic phase are always periodic with period 
two, oscillating either between a distinct pair of upper 
and lower values for each overlap component or between 
the same pair of values for all components, depending on 
the parity of c and on the state in the (a, T, v) phase di- 
agram. We argue that the cyclic behavior of period two 
is a property that follows essentially from the nature of 
the interactions, that is, from the strong symmetric se- 
quential term rather than being an artifact of the model 
due to the lack of lateral connections between the units 
or having binary units. First, in support of our claim, 
work in progress on the parallel dynamics of a fully con- 



nected recurrent network of binary units and patterns 
indicates, indeed, that there are exclusively cyclic states 
of period two in a finite region of the phase diagram [TtJ ■ 
Second, we checked explicitl y th at numerical simulations 
with threshold-linear units [l2j in our feed-forward lay- 
ered network yield only cyclic states of period two for 
typical low values of a and v, at T = 0. The specific 
regions of the phase diagram where distinct pairs of up- 
per and lower values or the same pair of values for the 
overlaps appear depends, of course, on the details of the 
model. 

The phase diagrams found in this work, are quite dif- 
ferent from those for asymmetric sequence processing 
0, [TB|. Indeed, the latter exhibit a v (1 — v) du- 
ality that appears in the form of symmetric phase di- 
agrams with a correspondence between fixed-point solu- 
tions for large v and stationary cycles of period c for small 
v. There is, apparently, no such duality in the case of 
symmetric sequence processing and whenever stable cy- 
cles appear they are of period two, independently of the 
number of condensed patterns c. Also, the strong c de- 
pendence of the stability of the cyclic phase is in contrast 
with the results for asymmetric sequence processing, in 
which the boundary of the cyclic phase practically does 
not depend on the number of condensed patterns p^.[l5|. 

Although the work presented here is restricted to a lay- 
ered feed-forward network with no lateral interactions, it 
is expected to exhibit further features of a recurrent net- 
work beyond those pointed out above, in particular, the 
robustness to both synaptic (T) and to stochastic noise 
(a) due to the non-condensed patterns over a sizeable 
ratio v/(l — v) of the relative strength of the Hcbbian to 
sequential interaction (Figs. 1(a) and 1(b)), respectively. 
The robustness becomes stronger in the case of decreas- 
ing even values of the number of condensed patterns c 
and weaker for odd values of decreasing c. 

The model used here has several limitations with re- 
spect to a closer to biological network which do not allow 
to make the proper quantitative predictions to compare 
with experiments, mainly the binary full activity units, 
in place of continuous or integrate and fire neurons, and 
unbiased (high activity) patterns, without lateral inter- 
actions between units in the same layer. Despite those 
limitations, there is the possibility of making extended 
qualitative predictions for the kind of visual-task experi- 
ments in the IT primate cortex and their interpretation in 
terms of correlated states in simple models for a recurrent 
network [8-12]. Those works provided a connection be- 
tween a fixed-point attractor dynamics in a recurrent net- 
work and persistent activity in a biological system. Our 
work suggests a connection between a periodic attractor 
dynamics in a recurrent network trained with patterns 
in a random order and with patterns in a sequence and 
a kind of oscillating persistent activity in the IT cortex 
with the specific cyclic behavior discussed in this work. 
The original experiments were based on intensive train- 
ing with patterns in a random order and with patterns 
in a sequence. One may argue that if the training of a 
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primate with visual patterns in random order, which is 
supposed to be a realization of a Hebbian rule, is not 
sufficiently strong, one may have a situation as that de- 
scribed here for small v with the presence of correlated 
cyclic states of period two, in which the correlation coef- 
ficients decay with increasing distance from the stimulus, 
up to a finite value. 

The results presented here should stimulate further 
theoretical and experimental work for the case of weak 
Hebbian reenforcement of patterns. It may also lead 
to interes ting applications in information processing in 
networks (l8| . 
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where {ui'j} is a set of independent Gaussian random vari- 
ables uj[ = Sp^/i £'i P Q P w it n mean = and width A 
defined by Eq. (6). Incidentally, the same expression for 
the embedding field without the need of separating a sin- 
gle term on the right may be used to obtain the recursion 
relation for the overlap components that yield Eq. (8). 

Turning now to the configurational average in Eq. 
(A3) over primed variables, it decouples into a product 
of averages. Making an expansion to leading order in 
Q% = 0{1/VN) we obtain 



tanh(/3#X) = tanhft&tf.Am) + C<) 
+PQ%[1 - tanh 2 pW&'.Am) + tfz<)] . (A.5) 

Averaging first with respect to the variable and then 
taking the configurational average over the Gaussian vari- 
able we obtain 



Unh(pCh'i)= K QN , ( A -6) 



APPENDIX: RECURSION RELATIONS 



in which K = (3 (1 — q), with q defined in Eq. (10). 
Substituting Eq. (A6) in (A3) we obtain, to 0{l/N), 



We present next an outline of the derivation of the re- 
cursion relations for all C n 's. It is based on an extension 
of the usual procedure [HI, [l6| adapted to our specific 
synaptic matrix. The main step is the recursion relation 
for the correlation between any two microscopic overlaps 
with the non-condensed patterns which enter in Eq.(9). 

We start with the definition of the average 



1 



'i' 



(A.l) 



for (j,, v = c + 1, . . . ,p, in which the primed (unprimed) 
variables refer to layer I + 1 (/), respectively. The aver- 
ages are explicitly calculated only with respect to primed 
variables since the averages over the underlying unprimed 
variables in the lower layer are taken care by means of 
the law of large numbers. Writing Eq.(Al) in the form 



(A.2) 

allows us to take the thermal averages leaving, in our case 
of binary patterns, 

(M# M'£ ) = % + -L ^tanh^M) tanh(/?^) , 



(A.3) 

using the fact that the patterns are uncorrelated variables 
with ^ = Sfiv The embedding field ^h! i is given by 

ifK = Cii' -Am) + Q% + tfj. , (A.4) 



{ m'>;m<»} = ^ + k 2 q» n q» n 



N 



We can now apply Eq. (9) to layer I 
it in terms of the correlations 



(A.7) 

- 1 and write 
with the aid 



of Eq. (7). Then, Eq. (A7) yields the p — c recursion 
relations 



A 

c; 2 



C' 2 



a + K 2 A 2 " 



46(1 



b)K 2 G\ 



2(1 - b) 2 K 2 [a/K 2 + A + C2 

2^2 



b K C( 
C\\ + (1 



26(1 - b)K 2 [a/K 2 
b) 2 K 2 [3Cf + Ci] 
b 2 K 2 C\ + 26(1 - b)K 2 [Cl + 
(l-b) 2 K 2 [a/K 2 + A 2 



2C 2 



cl. 



(A.8) 
(A.9) 

(A.10) 



up to 



CL 



b 2 K 2 C 2 +2b(l-b)K 2 [C 2 _ 1 
(1 - b) 2 K 2 [Cl„ 2 + 2C 2 + C 2 +2 } , (A.ll) 



in which n = 3, 4, . . . ,p — c — 1, and these equations have 
to be solved numerically. 

The extension to Q-state neurons and patterns, for 
Q > 3 is straightforward, as well as for continuous neu- 
rons. 
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